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Introduction 


of a design” 1 criteri^such^^that^he^im ° f t 8imulators is ‘he development 
emulate real world situations In . ?“ ulators can be adjusted until they 
obtained to compare two simulat-™- thl f pap f r 8uch a design criteria is 
or their equivalence 

can be considered as the same pr^bUm as ?he ° f tWO slmu letors 
ituation and a simulation's representation of “ parison of a r eal world 
The design criteria developed here involJ^ W ° rld situa tion. 

operator and defining simple parameters m ? delln 8 of the human 

ulator and in the real woPld TitZTiZ UtZ ** h±S behavlor ** the sim- 
human operator parameters to 1 the process of obtaining 

meaaurM are also obtained on these h° evaluate »i*ilntors, 

be used to describe the human as an info™*M characteristics which can 

Such modeling is motivated by the wo^r^r ?F? Ceasor and controller. 

in s a l ° therS ' FlrSt a stud y ia conducted on thi * / 6I J der8 f2] * Verplank 
in such a manner that this modeling aonroarh Z th v slmulator design problem 

criteria for the comparison of two^iSlatoPs ** U86d t0 develo P a 


Symbols 


ST(t) 

*(t) 

v(t) 


N 


Stick response of the human operator , 

The Kalman filter's best estimate of ST(t), 

The residuals, innovations, or modeling error. 

An approximation to signal power generated by the human. 
An approximation to noise 


power generated by the human. 
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Symbols 


e(t), E(s) 

* 

lo 


A,B,H 


XjUJ.XjU) 


P.R.Q 



BU 


The closed loop error signal and its Laplace transform 
The discrete state transition matrix. 

One standard deviation of a parameter estimate. 

The Kalman Gain matrix. 

The system gain matrices. 

State variables which describe ST(t) and ST(t). 
Unknown covariance matrices. 

Sample covariance function (from the data). 

Sample, normalized autocorrelation function. 

An approximation to bandwidth of the human operator. 




j 


THE SIMULATOR DESIGN PROBLEM 


in the comparison of two simulator, or in the comparison of a simulator 
with a real world situation, an assumption is made as folio . 

Asaunptlon^ lXl- stauUtot B u the „ UMn operator in Simulator A ha. th. .... 

"is ccrca srssss! : sj&ss.'Sir 

described by assumption (2) 

2SrS;.x: r JS fata sr-. 
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In practice, the usefulness of assumption (2) has application if it is 
possible to take data in the real world situation as well as in the simulator. 
If the simulator can be adjusted so that the human operator parameters in the 
real world situation are close to the human operator parameters in the simula- 
tor, then the simulator has replicated the real world situation. This agrees 
intuitively with the definition of a replication of an experiment. A repli- 
cation of an experiment is simply two empirical runs of data in which some 
variable shows consistency in both of the two empirical runs. In this case 
the variables that are to show consistency are the human operator model 
parameters. If these parameters show consistency between the real world 
situation and the simulator, then the simulator has replicated the real 
world situation. If the human operator appears the same in both the simulator 
and in the real world situation, and he rates the two to be the same subject- 
ively, then the simulator has reproduced the desired environment from the 
point of view of the human. 

The data base used to study the measures of simulator credibility 
involves a washout experiment as discussed in [4]. This experiment provides 
a unique opportunity to study how well the simulator replicates the real 
world situation. 


THE G-VECTOR TILT WASHOUT EXPERIMENT 


The G-Vector tilt washout experiment conducted at the Aerospace Medical 
Research Laboratory provides a data base to investigate the simulator 
credibility question. The data base used here involved a large centrifuge 
which has the capability cf positioning the roll axis normal to the earth's 
gravity (called 0°) or paralltl with the earth's gravity vector in which the 
subject is on his back or supine ^called 90°). Six experimental conditions 
were considered in this study. 

0° Conditions (Upright Position) 90° Condition (Subject on his back) 

0° Motion 90° Motion 

0° Washout - Attenuation only 
0° Washout - 1st. order washout 
0° Washout - 1st. order + attenuation 
0° Washout - 2nd. order 


As the subject makes a command stick response, the simulator rolls to Q 
simulate an aircraft in a banking maneuver. It is obvious that in the 0 
(upright) motion case the human has both tilt cue information as well as 
angular acceleration cue information. In the 90° (subject on his back) 
motion case, the human does not have the tilt cue information. The four 
washout conditions were conducted at 0° (upright position) and a washout 
circuit was installed between the stick response and the plant's roll 
characteristics (Fig. (la-b)). The effect of the washout circuit is to 
distort the motion cues to the human. 

The manner in which this data base is equivalent to the simulator cred- 
ibility problem is that the "real world" is defined as the 90° motion case. 
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Th e qiiestion Is then asked, which washout scheme at 0° is closest to the real 
00ti T ? a8e? The ° Wa8kout conditions contain reduced tilt cue 

Jirc^Us ThrldpH C ° ntaln , 80me dlstorted Botlon cues from the washout 
lence def ! J" 8 proced " e whlc * enables the determination of an equiva- 
lence definition between two simulators is presented next. 


THE MODELING APPROACH 


nicer 


i ^ U8tr ® tes how the modeling approach was conducted here, 
collected from the various experimental conditions a post 
thlt u 1 analyal ® wa ® conducted with a model developed in sucha manner 
WUh Tl?” C ox bC f deled 88 an information processor and controller. 

! ° fi8U f 6 (2 l* the input t0 the B0d ®l i® the time series e(t) 

^ err ° r 8lgnal) ‘ Thc P«Pc®* of this modeling appr^achisto 
choose model parameters such that the model’s output *(t) is an aecur!t. 

ZEST**" ° £ th * Bea8ured 8tic,t response of the human. The measure of 
which Mtisfiei- y cxpre88ed ln the residuals or output modeling error v(t) 


-5 o h : ss sswsffl.** thc « - • 

me»n ( v(t) ] - E(v(t)J - 0 (2 ) 

var [v(t)J - E[v(t)v T (x)J - R (3) 

structure 1. di.cu...d to d«crlbT U.W. wrest. 


A SIMPLE MODEL TO DESCRIBE HUMAN OPERATOR CHARACTERISTICS 


' h * r * ct " iz * th. hen oper.tor 

, ot interest for this study. From previous studies rs ai ».i.„ . 

sag 

“J°JJ •«“•! experimental P«—t.r. 

5 £SS-’ 


channel capacity - Bandwidth * log 1Q ( ^ ) 


S+N 

N 


( 4 ) 
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The Bandwidth tern is analogous to speed and the term log 1Q (-~) is analogous 

to accuracy. If the human operator has characteristics similar to an informa- 
tion channel, one would expect a product of the form of equation (4) to be 
invariant over several experimental conditions. It is then necessary to 
determine only two characteristics of the human operator in this representa- 
tion of responses. To determine bandwidth, use is made of the human describing 

function plots. In the determination of the accuracy measure, a Ka lman filter 
must be used. 

From figure (3), it is desired to have a method by which an approximate 
measure of human operator signal/noise ratio can be determined. In this 
modeling procedure, the Kalman filter is initially specified to have input- 
output characteristics similar to those obtained from the describing function 
with the addition of some phase lag to account for the time delay of the 
human. The unknown Kalman gain coefficients (which represent the uncertainty 
terms or covariance matrices) are updated [7] in such a manner that the 
residuals v(t) are white. The signal to noise ratio can then be approximated 

N 

S+N -v i-1 ISTftj)] 2 

~ * N (5) 

r (v(t.)j 2 
i-l 1 


It is noted that the variance of the residuals v(t) are a measure of human 
uncertainty with respect to the error signal. This is true because the 
Kalman filter output $(t) is that portion of the stick response correlated 
with the error signal. The residuals v(t) are that portion of the stick 
output not correlated with the error signal. This definition of human uncer- 
tainty differs from the classical definition of remnant [8,9] which is defined 
as that portion of human response not correlated with the input forcing 
function. This definition of human uncertainty is concerned with that part of 
the human response which is totally non-productive in reducing the error 
signal. This is easily seen to be true by noting that v(t) when passed 
through the plant and around the loop still is uncorrelated with the error 
signal. Hence it cannot constructively be used to reduce the error signal 
because of its orthogonality to it. This measure of human uncertainty is a 
true measure of human output not useful in the tracking task. Next, a descrip- 
tion of the measures of bandwidth and accuracy obtained from this modeling 
procedure are presented. * 


CALCULATION OF BANDWIDTH 


In the computation of a measure of the bandwidth of the human operator, 

rhlnnfi f i ni ^ * j Xlst attem P ti ng to treat the human as an information 
channel [10] . This is due to difficulties in determining the true describing 
function from measured data variables and the effects of correlation between 
the human s remnant response and portions of the measured error signal. In 
this paper several approximations will be made. Figure (4) illustrates t ie 
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describing function of the human for the 0 motion case* Across the six 
experimental conditions considered here, the shape of the human operator 
describing function remained essentially the same; the major change between 
experimental conditions was only due to the d.c. gain values where the 
describing function was at a maximum. The ensuing analysis was conducted on 
the spectrum generated by the target frequencies. The reason it is necessary 
to work with the target frequencies is that if the target forcing function 
were zero, the describing function of the human operator obtained from only 
the human operator response (or for small values of disturbance input) is 
Just equal to -1/plant. Tills result is well known [10,11]. 

From the target spectrums all experimental conditions are rated in order 
of their maximum gain value (table I) . From table T. it is seen that 0° static 
has the lowest gain value. The largest frequency tlve human will pass for this 
value of gain is now determined for each experimental condition. 

This definition of the human operator bandwidth is the highest 
frequency at which the human will respond with gain of 0.5 db. In other words 
across all experimental conditions, the range of frequencies (from 0.0 radians 
and upward) is obtained that the human will pass with gain greater than 0.5db. 

In this manner a normalization is conducted on one experimental condition ver- 
sus the remaining experimental conditions. This is a logical definition of 
human bandwidth and is one of many possible methods to approximate the bandwidth 
of a control system 112] . Measures of human uncertainty in tracking are. det- 
ermined next. 


Table I - Bandwidth Computation - Subject - Eric 


Experimental 

Condition 

Maximum Gain 
in db 

Bandwidth A Higest Frequency 
where gain >_ 0.5 db 

1C 

0° Motion 

6.5db 

10.8 Rad/Sec 

m 

Washout 

Attenuation only 

4.8db 

9.6 Rad/Sec 

0.3 

90® Motion 

4.5db 

9.5 Rad /Sec 

0.7 

Washout 1st Order 
+ Attenuation 

3.5db 

9.2 Rad/Sec 

1.1 

Washout 
1st Order 

2.9db 

8.3 Rad/Sec 

0.8 

Washout 
2nd Order 

3.3db 

8.2 Rad/Sec 

1.1 

0* Static 

0.5db 

7.3 Rad/Sec 

1.0 
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MEASUREMENT OF ACCURACY OR SUBJECT UNCERTAINTY 


With reference to figure (5) it is desired to update the model parameters in 
such a way that the innovations sequence v(t) is a white, random process. 

The method of updating the parameters is based on an algorithm [7] which is 
actually a maximum liklihood procedure. In this manner a unique value of e 
optimal gain can be determined which maximiz.es the probability density function 
of the structure of the assumed model based on all the available data points. 
The optimal gain is the principal part of the discrete Kalman filter model 
which is described by: 


S' 


1+1/ i 


e AT dxBcol[e(t),e(t)] (6) 


x i/i 


c i/i-l 


+ K [ z. 


" H %/i-i 1 


(7) 


where is the minimum variance estimate of the human's stick response. The 

matrix <|> is the discrete transicion matrix associated with the human's transfer 
function determined as folxows: 


Let ST[s] n d (s+a) 

E[s] (s+b) (s+c) 


< 8 ) 


i.e. a fit of one zero and vo poles is conducted on the human's transfer func- 
tion to the describing function data (Bode plot). The coefficients a,b,c, and d 
are adjusted to try to match the phase data as well as the magnitude data. Im- 
plicitly the human’s time delay has been included in the representation (8) 
through the adjustment of the parameters b and c. Future work will be done 
to 3 tudy more exact fits. The matrix $ is then determined via ^ 

where ^t*.04 seconds (the sampling rate) and the matrix A i3 determined via. 


e(t)j 

where 

x. (t) - ST(t) 

* 2 (t) - £ ST(t) j 

and equation (9) is the time domain representation of equation (8). The j 

matrix H in equation (7) is specified by H - [1,0]. The Kalman gain K q 

satisfies: 

K » P H T (HPH T + R)" 1 
0 

P - 4 [P-PH T (HPH T + R) _1 HP]* T + Q 
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where the covariance matrices Q and R describe the human’s uncertainty in 
the tracking task. The manner of obtaining the Q and R matrices is based on 
the algorithm in [7]# Initial matrix values denoted as and are chosen* In 

order to establish the updating rule, it is necessary to define the sample cov- 
ariance function* 


Let: 


^ 1 N T 

C k - N Z Vi-k 

i= k 


is a sample covariance function. The matrices R and Q are now updated [7] via: 


where 


\ - "V T > _ 

p /- k „ % + a * fe ! 


where 

and 


A 

A 


(a t a)" 1 a t 

|_h*(i-khH 




and finally Q Is determined via: 


Q k - P - - 4> (I-KH)P (I-KH) T (j» T 

This algorithm has been shown to converge [7] and is equivalent to maximizing 
the log-likelihood function of the model structure conditioned on the data. 

The final validation of this modeling effort is the need to test the 
residuals for whiteness. To accomplish this goal the normalized auto correla- 
tion function ^ is computed as follows: 



The test of whiteness of the residuals is a 95% whiteness test on ft . The 

k 

95% confidence limits for are 1.96/ MTT where N is the number of samples. 

The band + 1.96 /Mn is constructed about zero. If less than 5% of the sample 
points lie outside the band, the sequence is white. If more than 5% of the 
sample points lie outside the band, then a significant correlation exists in 
the residuals and the sequence is not white. Figure (6) illustrates the 
sample auto-correlation function obtained here from the data after the 
residuals have been whitened via this algorithm. 






/ 


RESULTS FROM THIS ANALYSIS 


Figure (7) represents the type of diagram obtainable from this type of 


510 


analysis procedure. The vertical axis Is a plot of the measure of bandwidth 
as shown in Table I. The horizontal axis indicates numerical values of 
the accuracy measure or S/N ratio obtained here. Also plotted is the curve 
of constant capacity based on this analysis procedure. The numerical values 
resulting from this investigation of the data are given in Table 11: 

Table II - Speed - Accuracy Results 


Exp 

Condition 

Mean 

BW 

lo 
of BW 

N=Mean 

log 10 S/N 

la 

l°g 10 S/N 

Mean Capacity 
BW*log 10 (l+S/N) 

0° motion 

10.8 

2.0 

3.265 

.191 

34.346 

Washout 

Attenuation only 

9.8 

0.3 

3.278 

.088 

31.719 

90° motion 

9.5 

0.7 

3.358 

.089 

31.5025 

Washout 1st order 
+ Attenuation 

9.2 

1.1 

3.378 

.064 

30.79 

Washout 
1st Order 

8.3 

0.8 

3.373 

.043 

27.813 

Washout 
2nd Order 

8.2 

1.1 

3.412 

.082 

27.634 


Also plotted in figure (7) is the invariant rule: 

BW * log 10 (l+S/N) ■ Constant - 30.6 (11) 

The constant 30.6 is the mean of the values of capacities obtained in the 
right most column in Table II. From figure (7) it is noted that most of the 
experimental conditions fall near this line. 

Figure (7), by itself, is the diagram which can be used to assess, the 
fidelity of a simulator in comparison to the real world data. If 90° motion 
is considered the real world situation, the washout scheme closest (distance 
wise) to this situation is 1st. order + attenuation. The other washout 
schemes are sucessively further away in this diagram and therefore, further 
from reality. The reason why it is said that the two experimental conditions best 
replicate each other is that the human exhibits almost the same bandwidth 
(or speed characteristics in tracking) and almost the same uncertainty 
characteristics (as measured by the S/N ratio) . 

Another interpretation of figure (7) is to consider the inverse problem 
associated with modeling; i.e. given the model parameters, can an 
analog simulation be built which will recreate the original empirical data. 

If the human in the loop were replaced by a quantitative description (e.g. 



bandwidth and S/N ratio), the analog simulations of the 90 motion case and 
washout 1st order + attenuation would most closely replicate one another. This 
is true because the only difference between the two simulations would be the 
parameters which describe the human operator. If these parameters are close 
to one another in some sense, then these simulations would best match. This 
is the motivation for using figure (7) to study simulator fidelity. 

One additional comment needs to be made about why the washout scheme of 
1st. order + attenuation best matched the 90° motion case. The 0° washout 
condition provided tilt cue information but the 1st. order + attenuation 
washout filter phase lag had the effect of distorting these tilt cues 
sufficiently to replicate the 90° motion case. For the case of attenuation 
only, the tilt cue had an effect closer to 0° motion (as expected). Also, as 
the washout scheme added more phase lag (2nd order case) , the deviation from 
reality became more pronounced and the human operator dropped his bandwidth 
accordingly. 


Future Research 


The primary approximation used here was in the evaluation of human operator 
bandwidth. This approximation also effected the S/N ratio because the A matrix 
in the Kalman filter depends on this approximation. Future research will 
consider more accurate methods of evaluating bandwidth and including human 
time delay. In addition, a comparison will be made in the information rate 
obtained here to results from discrete tasks ( approximately 3.0 bits/sec [13]) 
and to other information measures obtained from vision [14], reading [15], 
and control systems in general [16]. Another approximation utilized here was 
that the S/N ratio of the human was assumed to be constant over the entire 
frequency spectrum. In [3] the analysis procedure was able to study the 
capacity measure across the entire frequency spectrum. The procedure consid- 
ered here can be extended in this respect. Also, since the analysis conducted 
here only involved one subject, future work will consider this analysis across 
different subjects, and use will be made of these measures of human invariance 
and subjective uncertainty in various task situations. 


SUMMARY AND CONCLUSIONS 


A study of design rules for the evaluation of a simulator's fidelity to 
the real world situation was conducted. The measures of model parameters 
obtained here give rise to information-theoretic models of the human operator. 
It appears that an invariant rule may exist on the human's ability to do 
information processing over a variety of different experimental conditions. 
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FIGURE (la) -90° Motion Tracking 



FIGURE (lb) -0° Washout Circ u i t 
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FIGURE (4) - The Human Operator Describing Function 
0° Motion Case 



FIGURE (5) - Calculation of Subject Uncertainty 







Figure (6) - R( x ) versus 
Iteration 5 - The Sample 
Residuals are White For 
N-1200 data points. 











